Fine Grain Synthetic Educational Data: Challenges and Limitations of Collaborative Learning Analytics

While data privacy is a key aspect of Learning Analytics, it often creates difficulty when promoting research into underexplored contexts as it limits data sharing. To overcome this problem, the generation of synthetic data has been proposed and discussed within the LA community. However, there has been little work that has explored the use of synthetic data in real-world situations. This research examines the effectiveness of using synthetic data for training academic performance prediction models, and the challenges and limitations of using the proposed data sharing method. To evaluate the effectiveness of the method, we generate synthetic data from a private dataset, and distribute it to the participants of a data challenge to train prediction models. Participants submitted their models as docker containers for evaluation and ranking on holdout synthetic data. A post-hoc analysis was conducted on the top 10 participant’s models by comparing the evaluation of their performance on synthetic and private validation datasets. Several models trained on synthetic data were found to perform significantly poorer when applied to the non-synthetic private dataset. The main contribution of this research is to understand the challenges and limitations of applying predictive models trained on synthetic data in real-world situations. Due to these challenges, the paper recommends model designs that can inform future successful adoption of synthetic data in real-world educational data systems

Early-warning prediction of student performance and engagement in open book assessment by reading behavior analysis

Digitized learning materials are a core part of modern education, and analysis of the use can offer insight into the learning behavior of high and low performing students. The topic of predicting student characteristics has gained a lot of attention in recent years, with applications ranging from affect to performance and at-risk student prediction. In this paper, we examine students reading behavior using a digital textbook system while taking an open-book test from the perspective of engagement and performance to identify the strategies that are used. We create models to predict the performance and engagement of learners before the start of the assessment and extract reading behavior characteristics employed before and after the start of the assessment in a higher education setting. It was found that strategies, such as: revising and previewing are indicators of how a learner will perform in an open ebook assessment. Low performing students take advantage of the open ebook policy of the assessment and employ a strategy of searching for information during the assessment. Also compared to performance, the prediction of overall engagement has a higher accuracy, and therefore could be more appropriate for identifying intervention candidates as an early-warning intervention system

Fostering Evidence-Based Education with Learning Analytics: Capturing Teaching-Learning Cases from Log Data

Evidence-based education has become more relevant in the current technology-enhanced teaching-learning era. This paper introduces how Educational BIG data has the potential to generate such evidence. As evidence-based education traditionally hooks on the meta-analysis of the literature, so there are existing platforms that support manual input of evidence as structured information. However, such platforms often focus on researchers as end-users and its design is not aligned to the practitioners’ workflow. In our work, we propose a technology-mediated process of capturing teaching-learning cases (TLCs) using a learning analytics framework. Each case is primarily a single data point regarding the result of an intervention and multiple such cases would generate an evidence of intervention effectiveness. To capture TLCs in our current context, our system automatically conducts statistical modelling of learning logs captured from Learning Management Systems (LMS) and an e-book reader. Indicators from those learning logs are evaluated by the Linear Mixed Effects model to compute whether an intervention had a positive learning effect. We present two case studies to illustrate our approach of extracting case effectiveness from two different learning contexts – one at a junior-high math class where email messages were sent as intervention and another in a blended learning context in a higher education physics class where an active learning strategy was implemented. Our novelty lies in the proposed automated approach of data aggregation, analysis, and case storing using a Learning Analytics framework for supporting evidence-based practice more accessible for practitioners

La technologie du livre électronique pour faciliter l'enseignement universitaire pendant la COVID-19 : Expérience Japonaise

UNESCO reported that 90% of students are affected in some way by COVID-19 pandemic. Like many countries, Japan too imposed emergency remote teaching and learning at both school and university level. In this study, we focus on a national university in Japan, and investigate how teaching and learning were facilitated during this pandemic period using an ebook platform, BookRoll, which was linked as an external tool to the university’s learning management system. Such an endeavor also reinforced the Japanese national thrust regarding explorations of e-book-based technologies and using Artificial Intelligence in education. Teachers could upload reading materials for instance their course notes and associate an audio of their lecture. While students who registered in their course accessed the learning materials, the system collected their interaction logs in a learning record store. Across the spring semesters from April - July 2020, BookRoll system collected nearly 1.5 million reading interaction logs from more than 6300 students across 243 courses in 6 domains. The analysis highlighted that during emergency remote teaching and learning BookRoll maintained a weekly average traffic above 1, 900 learners creating more than 78, 000 reading logs and teachers perceived it as useful for orchestrating their course.L'UNESCO a signalé en mars 2020 que 84, 5 % du total des étudiant·e·s inscrits sont affectés d'une manière ou d'une autre par la pandémie de COVID-19, avec plus de 166 fermetures d'écoles à la grandeur de ces pays (UNESCO, 2020). Le Japon a lui aussi imposé un enseignement et un apprentissage à distance d'urgence, tant au niveau des écoles que des universités. Dans cette étude, nous nous concentrons sur une université nationale du Japon, et nous examinons comment l'enseignement et l'apprentissage ont été facilités pendant cette période de pandémie en utilisant une plateforme de livres électroniques, soit la plateforme BookRoll. En tant qu'outil externe, BookRoll a été relié au système de gestion de l'apprentissage de l'université. Cette initiative a également renforcé la volonté nationale japonaise d'explorer les technologies basées sur les livres électroniques et d'utiliser l'intelligence artificielle (IA) dans l'enseignement. Les enseignant·e·s pouvaient télécharger du matériel de lecture, par exemple leurs notes de cours, et y associer un enregistrement audio de leur prestation. Pendant que les étudiant·e·s inscrits à leur cours accédaient au matériel d'apprentissage, le système collectait leurs interactions dans un registre d'apprentissage. Au cours des semestres du printemps, d'avril à juillet 2020, le système BookRoll a recueilli près de 1, 5 million d’interactions concernant les lectures de plus de 6 300 étudiant·e·s dans 243 cours de 6 domaines, avec plus de 1 900 apprenant·e·s qui avaient créé plus de 78 000 entrées de journal, en mode lecture, par semaine. Bien que ce soit les cours de sciences et d'ingénierie qui ont principalement utilisé la plateforme, les cours de droit et d'études linguistiques l’ont utilisée pour y déposer des enregistrements audio associés à des documents à lire. L'analyse des interactions des étudiant·e·s avec le contenu a révélé que les actions d'apprentissage actif, telles que l'utilisation d'annotations sur le texte, étaient plus fréquentes dans les cours de sciences humaines. Enfin, des recommandations ont été formulées sur la base de l'analyse et de la perception des enseignant·e·s sur l'enseignement et l'apprentissage à distance d'urgence en utilisant le système BookRoll pour orchestrer leur cours

Learning log-based automatic group formation: system design and classroom implementation study

Collaborative learning in the form of group work is becoming increasingly significant in education since interpersonal skills count in modern society. However, teachers often get overwhelmed by the logistics involved in conducting any group work. Valid support for executing and managing such activities in a timely and informed manner becomes imperative. This research introduces an intelligent system focusing on group formation which consists of a parameter setting module and the group member visualization panel where the results of the created group are shown to the user and can be graded. The system supports teachers by applying algorithms to actual learning log data thereby simplifying the group formation process and saving time for them. A pilot study in a primary school mathematics class proved to have a positive effect on students’ engagement and affections while participating in group activities based on the system-generated groups, thus providing empirical evidence to the practice of Computer-Supported Collaborative Learning (CSCL) systems

Evidence Mining Using Course Schedule

Creating evidence from learning big data has become increasingly important as we can use eLearning infrastructure and store learning log digitally. On the other hand, we need to time and effort to create evidence because it is manual. In this paper, we proposed the method to make evidence easier. Especially, we focus on procedure to automatically select the duration of intervention and comparison data based on the course schedule information. We simulated the procedure and confirmed the making a case based on course schedule information. In the discussion part, we mentioned the points that should be further improved for practical use in the future. Through our method, we will democratize the evidence-based practice to all the teachers in schools

E-book-based learning activity during COVID-19: engagement behaviors and perceptions of Japanese junior-high school students

Recent spread of the COVID-19 forces governments around the world to temporarily close educational institutions. In this paper, we evaluated learning engagement, level of satisfaction and anxiety of e-book based remote teaching strategy on an online learning platform. The research involves 358 students at an urban junior-high school in Japan. Learning logs were analyzed to measure student engagement, whereas survey responses indicated their perception regarding the remote learning experience. Log analysis revealed that the average completion rate over 267 learning materials was 67%. We also observed a significant decrease in engagement 3 weeks after remote learning and different subjects and grades. Survey analysis showed students felt both satisfaction and anxiety about remote learning. However, there were significant differences in the level of satisfaction between different grades. The results indicated that (1) maintaining students' motivation is a challenge to remote learning in secondary schools, and (2) we need to relieve students' anxiety about their own progress in the class and their classes after the break. This study is the first to report trends in actual teaching-learning engagement, which were recorded during sessions of emergency remote teaching in Japanese schools. The results can inform the future implementation of remote learning in junior-high schools

Learning Dialogues orchestrated with BookRoll: Effects on Engagement and Learning in an Undergraduate Physics course

With COVID-19 pandemic forcing academic institutions to shift to emergency remote teaching (ERT), teachers worldwide are attempting several strategies to engage their learners. Even though existing research in online learning suggests that effectiveness of the online session is more dependent on pedagogical design rather than technology feature, teachers may still focus on the intricacies of the technology. In this paper, we present the evolution of an active learning pedagogy, supported by technology (eBook reader—BookRoll, Analytics Dashboard—LAViEW), for an undergraduate physics classroom across a semester that was affected by the lockdown due to pandemic. The technology-enhanced pedagogy evolved in three phases—technology used in “Content Focus” mode, technology used in “Problem Focus” mode and technology used in “Learning Dialogue Focus” mode. The entire activities were designed and implemented within the technology-enhanced and evidence-based education and learning (TEEL) ecosystem, which supported integration of learning technologies with analytics system. Comparison of the student’s learning logs indicated that there was a sustained engagement in the learning activities conducted during the blended (before lockdown) and online mode (during lockdown). We had conducted one-way ANOVA to compare the post-test scores for each teaching phase and found statistically significant differences in the latter phases. A preliminary qualitative analysis of the learner artifacts generated as memos in BookRoll during each phase revealed that students were posing conceptual clarifications during the latter phases. These were also having greater alignment with the session agenda and showed construction of new knowledge based on the seed knowledge provided during the instructor–learner interaction sessions. The study provides key insights into how reflection and practice by both learner and teacher improves the acceptance of technology-enabled pedagogy

Towards Predictable Process and Consequence Attributes of Data-Driven Group Work: Primary Analysis for Assisting Teachers with Automatic Group Formation

Data-driven platforms with rich data and learning analytics applications provide immense opportunities to support collaborative learning such as algorithmic group formation systems based on learning logs. However, teachers can still get overwhelmed since they have to manually set the parameters to create groups and it takes time to understand the meaning of each indicator. Therefore, it is imperative to explore predictive indicators for algorithmic group formation to release teachers from the dilemma with explainable group formation indicators and recommended settings based on group work purposes. Employing learning logs of group work from a reading-based university course, this study examines how learner indicators from different dimensions before the group work connect to the subsequent group work processes and consequences attributes through correlation analysis. Results find that the reading engagement and previous peer ratings can reveal individual achievement of the group work, and a homogeneous grouping strategy based on reading annotations and previous group work experience can predict desirable group performance for this learning context. In addition, it also proposes the potential of automatic group formation with recommended parameter settings that leverage the results of predictive indicators

LAView: Learning Analytics Dashboard Towards Evidence-based Education

The 9th International Learning Analytics and Knowledge (LAK) Conference : March 4-8, 2019, Tempe, Arizona, USALearning analytics dashboards (LAD) have supported prior finds that visualizing learning behavior helps students to reflect on their learning. We developed LAViEW, a LAD that can be easily integrated with different learning environments through LTI. In this paper, we focus on the context of eBook-based learning and present an overview of the indicators of engagement that LAView visualizes. Its integrated email widget enables the teacher to directly send personalized feedbacks to selected cohorts of students, clustered by their engagement scores. These interventions and dashboard interactions are further tracked to extract evidence of learning